posted 09-15-2006 01:38 PM            

What do you think? In my opinion, it is a major step forward, but it still needs some work as it leaves unanswered questions (unless I'm missing something).

Before I make my points, let me meddle on a side issue. Note in 3.12.1.2 it says "As with any polygraph examination... polygraph examinations do not take the place of an investigation," and it refers to the background investigation being necessary. We had this conversation before, but where is the "investigation" in a fidelity case? (I'm not opposed to those. I'm just wondering how we argue consistency in what we do.) Okay, enough meddling, and back to the issues.

In 3.12.9.1 it recommends any SRs on a multi-issue test be broken out and tested in a specific-issue test. I agree with that, and that is what ASTM says.

Note in 3.12.10.1 and 2 we're told multi-issue tests are less accurate than single-issue tests. True.

That means SRs on a multi-issue test may not be due to deception (because the error rate increases). Why then, does the policy refer to such SRs as "deceptive issue[s]" in 3.12.10.3? In 3.12.10.4 rather than calling an SR finding a "deceptive issue," it calls the SR question(s) "the issue of concern."

In 3.12.11.6 we're instructed to call a person "deceptive" if he, among other things, has SRs on a multi-issue (or specific-issue) test.

Think about that for a minute. I could run a multi-issue screeing exam and have an examinee react to, let's say, the drug question. I interrogate and he assures me there's nothing else out there, which is possible we all know.

I report him as "deceptive to the testing process." After calling him deceptive do I also recommend a follow-up specific-issue test to try to "resolve" the deceptive issue because, we know, the person may not be lying to that issue? If so, and that is reality, that would make me look foolish. Again, "He's 'deceptive,' but I don't know if he's deceptive, so I recommend more testing" is what I'd be reporting when all is said and done.

posted 09-16-2006 05:14 PM            

3.12.10 Use of a Diagnostic or Successive Hurdles Approach.

3.12.10.1 A multi-issue test format must be utilized in most pre-employment polygraph examinations because the examinations usually cover many areas of concern, i.e. theft, drugs, etc. Scientific studies have indicated that the more issues covered in an examination, the more likely that the accuracy of the examination will be affected in a negative way.

3.12.10.2 The most accurate testing format is a single-issue comparison technique.

3.12.10.3 Consequently, when using a multi-issue screening format, an examiner may need to administer a “follow-up” validated specific-issue examination to resolve any deceptive issues on the multi-issue test format. A specific-issue examination should be administered only if it is determined that the specific-issue examination will resolve the deceptive issue(s) from the multi-issue examination

3.12.10.4 The follow-up examination should be focused on the issue of concern identified in the screening exam.

3.12.10.5 The follow-up specific-issue examination should be scheduled for a later date.

3.12.10.6 If a follow-up specific-issue examination is recommended, the hiring agency shall decide if the follow-up examination will be administered.

Notice this last portion. It says, in essence, given the scenario I described above (deceptive, but maybe not due to deception) I now should wait for the agency to tell me if I'm done testing. I'm not done testing until I have resolved every issue with a confession, DI, NDI or NO. Why would I wait for the agency to tell me how to conduct a polygraph test?

Here's the other portion:

3.12.11 Test Evaluation

3.12.11.1 After the polygraph examination has been administered, the examiner will evaluate the charts using the appropriate method for the specific testing format being used, looking for any consistent, significant responses physiological to relevant questions that are timely to the questions.

3.12.11.2 If there are consistent, significant physiological responses noted to any of the relevant questions, the examiner will render the appropriate evaluation of the examination.

3.12.11.3 If sufficient criteria do not exist to render an opinion, the examiner shall evaluate the exam as No Opinion (NO) or Inconclusive (INC).

3.12.11.4 Additional testing may follow any of the above opinions.

3.12.11.5 If there are no consistent, significant physiological responses noted to relevant questions, the examiner should indicate such on the final report

3.12.11.6 After a confession, an admission, or a finding of consistent physiological reactions to relevant questions on any specific or multi-issue examination, the examiner shall call an applicant “deceptive” to the testing process.

posted 09-17-2006 10:09 AM            

quote:

---

3.12.10.1 A multi-issue test format must be utilized in most pre-employment polygraph examinations because the examinations usually cover many areas of concern, i.e. theft, drugs, etc. Scientific studies have indicated that the more issues covered in an examination, the more likely that the accuracy of the examination will be affected in a negative way.

---

I appreciate the spirit of empirical integrity of this, though I don't like the way its languaged here.

I do not like references to research without a citation to the authoritative reference... (maybe this is just a minor pet peeve of my own, from sitting in too many meetings with graduate students, neophytes, under-prepared trainers, and agenda-driven policy-wogs, who say vague things like "research shows..." without providing a reference or citation). Saying things like “research says...” or “studies show...” is a common form of manipulative argument, by persons who can't win the argument via there own logic and information, that places the discussion out of the reach those present who cannot question, challenge, examine or discuss with the data or authors of those un-cited authoritative studies.

I would prefer to make the point more generally, though just as accurately, with some version of this...

Multiple issue examinations offer improved sensitivity to a broader range of possible concerns, with an anticipated corresponding degradation of diagnostic specificity.

Adding multiple testing issues is both useful and desirable, because it makes the the more sensitive to the range of issues we are concerned about. If were not the case, then we wouldn't do it. To say that 'multiple issues affects accuracy negatively' represents an incomplete understanding or discussion around test accuracy issues. Sensitivity is a type of accuracy - its just not the same as diagnostic accuracy. We really should be teaching ourselves to understand this. Some of our smarter consumers and opponents already see these ambiguities, and ask things like "why do we do this" and "should not all tests be single issue." We use mixed issues tests because they offer improved accuracy (sensitivity) not degraded accuracy.

The statement above is like saying that rifles are more accurate than shotguns. Well, yes, maybe depending on the situation. Try hunting quail with your scoped .30/06. The shotgun is the firearm of choice because of its accuracy in getting the job done. On the other hand the shotgun is not much good with a whitetail deer at 200 yards. Real-world accuracy is what gets the job done.

The same concern occurs in polygraph and neuropsychological testing. In neuropsych testing we use batteries (mixed issues) of screening tests that we call "quick and dirty" because they are expedient and they work. However, they don't tell us much about the exact nature of a person's neurological deficits. Testing is like fishing with a net o spear. Cast a net (mixed issue test) and your going to catch all kinds of things. You might also have too coarse of a net, that allows small fish to swim right through - leading you to the erroneous conclusion that there are no fish. Or the net might be to small (blunt) for the great big and wily fish. So, you get the spear because of its pin-point accuracy - only now you had better know exactly where and what you are after, else no fish for you.

Neuropsych tests have the same limitations. Some are to coarse or blunt to detect real though very specific deficits (false negative conclusion). Some are so specific that they overlook deficits they are not intended to detect.

Screening tests are intended to be sensitive, so that when someone passes it, they are done. Then, we go to the trouble and expense of doing diagnostic testing on the areas of the screening test for which a persons exhibited difficulties. Screening tests are often intended to slightly over-predict the presence of problems - intentionally - with he goal of reducing costly false-negative errors. False positive results on screening tests are corrected upon more specific diagnostic evaluation. (As Don Krapohl said at APA - false positive errors can be recovered at later testing, false negative errors cannot.) Diagnostic tests are intended to be very specific, as they often inform some kind of necessary activity or response (hospitalization, surgery, chemotherapy, etc.) Screening tests alone, are insufficient to require such action, and simply tell us whether, ideally, there is likely not a problem.

We should be careful about not tripping ourselves with our inarticulateness around complex empirical concerns.

My point is that multiple issue tests offer an improved form of accuracy - sensitivity to an array of concerns. I would guess that some of those un-cited studies may have been shortsighted attempts to evaluate the diagnostic accuracy of mixed issue tests. But, without a citation, we have a very difficult time interpreting for ourselves how the merits and deficits of those particular studies might best inform this discussion.

So, Barry,

I think you are correct, in that we have perhaps not fully thought through how to use and discuss the results of screening and diagnostic tests. If we were positive that your subject were “deceptive,” no further testing would be necessary – but an interrogation might be in order. We could start to look kind of silly if we're not careful how we incorporate the conceptual language of testing. What you are really illustrating is that we may be engaging in some over-use of the terms and concepts of “deception” and “deceptive.” There is an important balance to achieve here, because emphasizing the language of deception is desirable in that it empowers leverage over the deceptive examinee. However, it is also important for us to avoid formulating our working conceptualization of “deception” in overly-materialistic terms – as if “deception” and “deceptive reactions” are physical things themselves. This is the reason that terms such as “significant reactions indicative of deception” become very important – they keep us thoughtful about the actual meaning of test results.

It looks a little like 3.12.11.x is an earlier or simpler version, and that 3.12.10.x (successive hurdles) was kludged onto the document. It would be better for the two sections to be reduced to one section that does a clearer job of addressing the results of screening and successive hurdles testing approaches.

I'm also concerned about this...

quote:

---

3.12.9.2 The multi-issue examination should be limited to not more than five (5) relevant questions. If more issues need to be explored, a second and/or third series should be administered.

---

Five relevant or target issues presents a complex set of mathematical problems, when considering the results of individual test questions. We not only increase the likelihood of deceptive results (by looking in more places) but we also compound the likelihood of inconclusive results.

Here is an illustration table

Distinct Targets 1 2 3 4 5

Inc rate
0.1 0.1 0.19 0.27 0.34 0.41
0.05 0.05 0.1 0.14 0.19 0.23
0.025 0.03 0.05 0.07 0.1 0.12

Resolution rate
0.9 0.9 0.81 0.73 0.66 0.59
0.95 0.95 0.9 0.86 0.81 0.77
0.975 0.98 0.95 0.93 0.9 0.88

The possibility of inconclusive results is compounded by the number of distinct questions – the old wisdom of 10% inc applies to single issue diagnostic tests. In mixed issues testing the inconclusive rate is more accurately estimated as the ideal inconclusive rate (10%) raised to the power of the number of distinct scorable target issues (relevant questions). There will always be inclusive test results, due in part to random measurement error and other variability in psychophysiology and conceptual language.

So, we might anticipate, with five questions, up to 41 percent inconclusive results. This is not really a problem as long as policies accurately reflect how to parse and respond to those test results. Successive hurdles and medical models do that.

However, high rates of inconclusive results presents a problem similar to that in PCSOT testing, in which consumers are then unhappy with the examiner (who now has a professional credibility and marketing problem due to all those blasted inconclusive results) who responds by interpreting increasingly marginal quality data in attempt to reduce inconclusive results. Based upon the “Garbage-in, garbage-out” principle, we might anticipate that test results become less reliable. To me this is both and empirical and ethical concern. I bet we can all think of examiners who don't have inconclusive results, and consumers would certainly prefer to purchase “confidence” than the uncertainty that actually comes with measurement and probability science.

The other thing that is not clear to me is the term “series.” If “series” is synonymous with “charts,” then it seems that one chart may be completed for each set of test questions. I don't take it this way, but some might, and perhaps RI testing formats don't always require more than one chart??? This should be clarified further. Perhaps some additional language such as “ a second and/or third series should be administered, with an appropriate number of test charts per the selected technique.”

R

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 09-20-2006 05:32 PM            

When a follow up is done on that issue, a single issue test, if the score shows deception we would then say he is D.I.. If the score indicates we say N.D.I.

If he was D.I. on an issue we no longer care if he was truthfull or deceptive to the other issues.

Jack

posted 11-03-2006 01:26 PM            

I've been playing with significance tests lately, and pondering things like two-stage decision rules and Bonferonni corrections to alpha thresholds.

In most research it is incorrect to conduct multiple significance tests. Instead we use multiple analysis of variance for all sets at once. The problem in polygraph is that we don't use analasis of variance across multiple RQs, and instead evaluate each RQ as a separate data point (what staticians call a point-estimate).

This seems necessary because multipe analysis of variance procedures cannot point to the issue of difference, it only says that the points/groups are different. Failing to reject the null-hypothesis on MANOVA (which says that the points/groups/RQs will not differ significantly (which means INC/NO in polybabble), and usually results in a series of regressions, significance tests, or other discriminate tests designed to isolate the issue of difference.

The problem occurs with the alpha (preset significance level). Conducting multiple parallel signficance tests increases, through chance, the probability of finding something significant - effectively inflating the alpha. Uncorrected inflated alpha can be estimated as the specified alpha multiplied by the number of separate tests - which is essentially the number of separate RQs (distinct questions scored as separate spots).

This underlies, in part, what Barry talks about when he says that screening tests lack sensitivity or specificity. In part this has been a problem because we have not thought it all through in a manner consistent with what we know about inferential statistics.

With 3 RQs and alpha set at .05 - employing the spot scoring rule effectively inflates the Alpha for DI test to .15 - because any DI question fails the test.

With 4 RQs uncorrected .05 alpha becomes .20

Now imagine the uncorrected inflation with at test that includes an uncorrected alpha of .05 and as many as five RQs (as allowed by the APA standards for screening exams) - the uncorrected inflated alpha with five RQs is effectively .25

Bonferonni suggested a cure for this problem of inflated alpha, and adjusts the alpha by dividing by the number of sets/groups/points - in our case the # of RQs. So a one-tailed alpha of .05 becomes a bonferonni corrected alpha of .0167 with three questions.

It may be appropriate to set alpha higher to begin with. It is appropriate to use lower alpha when the cost of a type 1 error (false alarm) is high, and lower alpha when the cost of a type 2 error (failed alarm) is high. As Don Krapohl says, we can recover from a false positive at subsequent testing or investigation, we cannot recover a false negative.

NDI results do not require Bonferonni correction - the null-hypothesis for NDI is that at least one question will be not-significant (no inflation of alpha). As stated earlier, the null hypothesis for DI is that all RQs will be not significant - meaning that any significant RQ means DI (which does inflate the alpha, and so requires Bonferonni).

So, even with significance tests, our decision schemes may not be symetrical, and it may be worth thinking about our tests with more consideration for testing objectives and the importance of clarifying our null-hypotheses.

Enjoy,

r

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 11-03-2006 11:12 PM            

You are talking statistics and I know you know what error is acceptable in that discipline.

posted 11-03-2006 11:31 PM            

------------------
"Gentlemen, you can't fight in here, this is the war room."
--(from Dr. Strangelove, 1964)

posted 11-04-2006 04:47 PM         

Also, are the exams administered to convicted sex offenders screening exams?

posted 11-05-2006 12:22 AM            

quote:

---

If someone goes INC or SR to a question in a screening exam and a break out exam is conducted on the INC or SR question, would you consider the break out exam to be a specific issue exam in the strict meaning of the definition? Maybe I'm off of base in my thinking, but even though you've narrowed the exam to one specific area of interest, the questions are still rather broad.

---

That's a really good question, and your point about the questions still being broad is well taken. The test is still a screening test, in the absence of a known event/direct allegation and reason to suspect involvement.

A look through the NRC/NAS report isrevealing. The authors favor the term event-specific over specific-issue. the term "specific issue" appears 8 times in the 416 page document while the term "event specific" can be found 25 times. This was not done without forthought - the term event-specific imposes a certain clarity about this very issue. We can have a single issue polygraph that is still a screening test and not a diagnostic test or investigative examanination. An Event specific test is just that.

------------------
"Deterrence is the art of creating in the mind of the enemy the fear to attack."
--(from Dr. Strangelove, 1964)

posted 11-05-2006 10:30 AM         

Also, DLC's as opposed to PLC wouldn't be a bad idea for pre-employment and employee screening exams. They're less confrontational, and easier to administer. Of course I can't speak for DLC's in the sex offender arena because it only seems reasonable that those exams are akin to criminal specific exams. Has anyone ever tried DLC's in that arena?

[This message has been edited by polypro (edited 11-05-2006).]